Skip to content

Make claude-review tolerant of missing structured_output#12

Merged
TimJohns merged 5 commits into
mainfrom
harden-claude-review-structured-output
Jul 25, 2026
Merged

Make claude-review tolerant of missing structured_output#12
TimJohns merged 5 commits into
mainfrom
harden-claude-review-structured-output

Conversation

@TimJohns

Copy link
Copy Markdown
Member

Problem

The claude-review job's analyze step (anthropics/claude-code-action with --json-schema) intermittently exits 1 with:

--json-schema was provided but Claude did not return structured_output. Result subtype: success

Result subtype: success means the compatibility assessment itself ran fine — only the machine-readable structured-output capture failed (anthropics/claude-code#23265). Because every GitHub Actions re-run is a fresh process (a fresh "first invocation"), re-running never escapes it, so this reds claude-review on Dependabot PRs across all caller repos even when the update is fine.

Bumping the action to v1.0.182 did not fix it — verified on epic-roadtrip-planner-webapp run 30139938013, which used claude-code-action@e0cf66d1 (v1.0.182) and still failed with the same error and no verdict.

Fix

Make the plumbing failure non-blocking while preserving the real verdict gate:

  • continue-on-error: true on the analyze step, so the action exiting 1 on a missing structured_output no longer fails the job.
  • The summary step now treats a missing/invalid structured_output as advisory — posts a note pointing at Claude's PR comment and exits 0 — but a valid object with a non-PASS recommendation still exits 1 and reds the check.

Net effect:

Case Before After
Cold-start plumbing miss (no structured output) ❌ red (spurious) ✅ green + advisory note
Real PASS verdict ✅ green ✅ green
Real FAIL / NEEDS REVIEW verdict ❌ red ❌ red (gate preserved)

The check is advisory anyway — the branch rulesets require only build (24.x) + workflow-lint.

Trade-off to weigh

When the assessment can't emit structured output, we now go green and rely on the posted PR comment for the human signal. If you'd rather the check stay red until the upstream action bug is fixed, don't merge — but note it reproduces on essentially every Dependabot PR today.

🤖 Generated with Claude Code

TimJohns and others added 2 commits July 24, 2026 19:20
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TimJohns

Copy link
Copy Markdown
Member Author

Live-tested on a real Dependabot PR

Tested this branch's build against epic-roadtrip-planner-webapp #1657 (jest-dom 6→7, a major → triggers review), run 30170333854:

  • claude-review job now succeeds (green).
  • ✅ The --json-schema … did not return structured_output error is gone, and the analyze step succeeded. That error is emitted by the action itself whenever --json-schema is set but nothing comes back, so its absence proves structured output was genuinely produced — the gate is real, not masked by the tolerance guard.

Root cause confirmed: it was track_progress: true forcing tag/comment mode, which never emits structured_output. Removing it puts the step in automation mode and the verdict comes back.

Caveat to decide: in automation mode the model returns the structured verdict but no longer posts the rich human-readable review comment — the verdict now lives in the job summary + the pass/fail gate. If we want the comment back, the follow-up is to add a summary field to the --json-schema and have the summary step post it as the PR comment.

TimJohns and others added 3 commits July 25, 2026 12:03
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TimJohns
TimJohns merged commit 9235560 into main Jul 25, 2026
1 check passed
@TimJohns
TimJohns deleted the harden-claude-review-structured-output branch July 25, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant